Wrong predictions can lead to various problems in di erent application domains, e.g., improper medical diagnosis [25] and tra c accidents [16]. Enhancing the DL application systems by reducing wrong predictions of DL models in producing outputs is desirable. Studies [9, 51, 52] have shown that DL models are vulnerable to operational input samples that can lead them to produce incorrect predictions in natural scenarios [52], and the prediction con dences of many such failing samples exceed those well-intended guarding con dence levels [54]. For example, strong sunshine may cause the camera of a self-driving car to capture an image full of white pixels, resulting in a prediction failure with high con dence. A major bottleneck in developing DL applications is detecting these overcon dent failures from their deployed DL application systems. To reduce unreliable predictions, many real-world machine-learning-based application systems are equipped with rejectors to discard uncertain decisions [17]. In DL application systems, many existing techniques [6, 17, 45] construct their rejectors for DL models to address the incorrect prediction problem. For example, many recent studies [2, 8, 42, 49] have been conducted to enhance the defense ability of DL models against out-of-distribution (OOD) samples from unknown classes or arti cial examples that are very likely to guide DL models to yield failures.

ROC or Receiver Operating Characteristic Curve is the most frequently used tool for evaluating the binary or multi-class classification model. Unlike other metrics, it is calculated on prediction scores like Precision-Recall Curve instead of prediction class. In my previous post, the importance of the precision-recall curve is highlighted as how to plot for multi-class classification. To understand ROC Curve, let's quickly refresh our memory on the possible outcomes in a binary classification problem by referring to the Confusion Matrix. ROC Curve is a plot of True Positive Rate(TPR) plotted against False Positive Rate(FPR) at various threshold values. It helps to visualize how threshold affects classifier performance.

Deep Recurrent Neural Networks (RNN) continues to find success in predictive decision-making with temporal event sequences. Recent studies have shown the importance and practicality of visual analytics in interpreting deep learning models for real-world applications. However, very limited work enables interactions with deep learning models and guides practitioners to form hypotheticals towards the desired prediction outcomes, especially for sequence prediction. Specifically, no existing work has addressed the what-if analysis and value perturbation along different time-steps for sequence outcome prediction. We present a model-agnostic visual analytics tool, HypperSteer, that steers hypothetical testing and allows users to perturb data for sequence predictions interactively. We showcase how HypperSteer helps in steering patient data to achieve desired treatment outcomes and discuss how HypperSteer can serve as a comprehensive solution for other practical scenarios.

In Build 2018, Microsoft introduced the preview of ML.NET (Machine Learning .NET) which is a cross platform, open source machine learning framework. Yes, now it's easy to develop our own Machine Learning application or develop a custom module using Machine Learning framework. ML.NET is a machine learning framework which was mainly developed for .NET developers. We can use C# or F# to develop ML.NET applications. ML.NET is an open source which can be run on Windows, Linux and macOS.

We build a deep reinforcement learning (RL) agent that can predict the likelihood of an individual testing positive for malaria by asking questions about their household. The RL agent learns to determine which survey question to ask next and when to stop to make a prediction about their likelihood of malaria based on their responses hitherto. The agent incurs a small penalty for each question asked, and a large reward/penalty for making the correct/wrong prediction; it thus has to learn to balance the length of the survey with the accuracy of its final predictions. Our RL agent is a Deep Q-network that learns a policy directly from the responses to the questions, with an action defined for each possible survey question and for each possible prediction class. We focus on Kenya, where malaria is a massive health burden, and train the RL agent on a dataset of 6481 households from the Kenya Malaria Indicator Survey 2015. To investigate the importance of having survey questions be adaptive to responses, we compare our RL agent to a supervised learning (SL) baseline that fixes its set of survey questions a priori. We evaluate on prediction accuracy and on the number of survey questions asked on a holdout set and find that the RL agent is able to predict with 80% accuracy, using only 2.5 questions on average. In addition, the RL agent learns to survey adaptively to responses and is able to match the SL baseline in prediction accuracy while significantly reducing survey length.